Handling flaky tests #732
Merged
Handling flaky tests #732
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## main #732 +/- ##
=====================================
Coverage 94.4% 94.5%
=====================================
Files 43 43
Lines 3434 3463 +29
=====================================
+ Hits 3245 3273 +28
- Misses 189 190 +1
|
khaeru
reviewed
Aug 2, 2023
There was a problem hiding this comment.
Looks effective, thanks.
Per the changes to the files, I think it would be better to do something like:
# Define a mark as a module global
FLAKY = pytest.mark.flaky(
reruns=5
rerun_delay=2,
condition="GITHUB_ACTIONS" in os.environ and sys.platform == "darwin",
reason="Flaky; See iiasa/message_ix#731"
)
…
# Use the mark to decorate multiple tests
@FLAKY
def test_add_bound_activity_up_all_modes(message_test_mp):
…Two reasons:
- A general heuristic (not sure where or from which example I picked it up) for pytest usage is to organize tests (modules, files, classes, and functions) by functional meaning or in a way that mirrors the organization of the code being tested. For instance, we might have a
class TestPlatformthat collects tests ofclass Platform. This makes it easy to navigate.
If these tests cease to be flaky (for instance, with a newixmp4backend) then the class grouping would no longer be meaningful and we'd have to move the tests out again. - It would create minimally-invasive, small diffs.
- It would be clear when multiple tests are marked in the same meaning, without having to read an compare the function arguments.
|
Thank you for the comment, I agree that your suggestion is more elegant. I will re-work this PR and the other ones. |
glatterf42
force-pushed
the
enh/ci/flaky-tests
branch
from
1316d3f
to
d65e36b
Compare
khaeru
approved these changes
Aug 4, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #731 by marking tests as flaky. Through discussion with @khaeru, we agreed that this is the best option for now; should the tests make more issues in the future, we can still identify their root cause.
How to review
PR checklist
[ ] Add, expand, or update documentation.Nothing new added.[ ] Update release notes.Nothing public-facing added